Spent most of the day talking to IT.
Lot of PC crashes. Negotiating this with IT.
Tried to get transfers working with our SToRM instance to see if there was any debugging info. After initial issue of an expired ca-trust certificate (bad certificate chain), fixed with:
wget https://linuxsoft.cern.ch/cern/centos/7/updates/x86_64/Packages/ca-certificates-2021.2.50-72.el7_9.noarch.rpm
sudo rpm -Uvh ca-certificates-2021.2.50-72.el7_9.noarch.rpmin ftsserver container (and on src-dev), was able to successfully transfer from this SE to itself:
[rbarnsley@src-dev ~]$ fts-rest-transfer-submit --access-token=`oidc-token ska-src-fts` -s https://srcdev.skatelescope.org/fts/rest https://srcdev.skatelescope.org/storm/sa/testfile https://srcdev.skatelescope.org/storm/sa/testfile2
Job successfully submitted.
Job id: 3851cab8-ddd6-11ec-a037-0242ac120004
[rbarnsley@src-dev ~]$ fts-rest-transfer-status --access-token=`oidc-token ska-src-fts` -s https://srcdev.skatelescope.org/fts/rest 3851cab8-ddd6-11ec-a037-0242ac120004
Request ID: 3851cab8-ddd6-11ec-a037-0242ac120004
Status: FINISHED
Client DN: 38d9270e-8d77-4e45-8ed0-b541be6c5e91
Reason:
Submission time: 2022-05-27T16:01:08
Priority: 3
VO Name: robbarnsleywhich leads me to believe that maybe i've been denylisted at the previous SEs?
Some useful commands (on src-dev):
fts-rest-whoami --access-token=`oidc-token ska-src-fts` -s https://srcdev.skatelescope.org/fts/rest
fts-rest-transfer-submit --access-token=`oidc-token ska-src-fts` -s https://srcdev.skatelescope.org/fts/rest davs://gfe02.grid.hep.ph.ic.ac.uk:2880/pnfs/hep.ph.ic.ac.uk/data/escape/ska/testing/da/06/ef1e45eb-009c-4501-923d-642dd3022c93 davs://amnesiac.cloud.cnaf.infn.it:8443/escape/ska/test-from-fts2
fts-rest-transfer-submit --access-token=`oidc-token ska-src-fts` -s https://srcdev.skatelescope.org/fts/rest https://srcdev.skatelescope.org/storm/sa/testfile https://srcdev.skatelescope.org/storm/sa/testfile2
fts-rest-transfer-status --access-token=`oidc-token ska-src-fts` -s https://srcdev.skatelescope.org/fts/rest Need to write this up still, and continue with run_fts_tests.py script on dev. Possibly contact the SE admins for the above storages too - see if there's anything their side (bit strange considering it authenticates OK with gfal though!)
Started writing script for FTS configuration. Previous SEs now not authenticating though. Debugging this.
Got a token transfer working with our instance of fts, gave up trying to get x509 working behind the proxy.
[rbarnsley@src-dev install]$ fts-rest-whoami -s https://srcdev.skatelescope.org/fts/rest --access-token=$ACCESS_TOKEN
User DN: 38d9270e-8d77-4e45-8ed0-b541be6c5e91
VO: robbarnsley
VO id: c8fd3c13-ff80-5f40-82fc-b9c3cbb1efcb
Delegation id: 8d68942b0d1b8d91
Base id: 01874efb-4735-4595-bc9c-591aef8240c9
[rbarnsley@src-dev install]$ fts-rest-transfer-submit --access-token=${ACCESS_TOKEN} -s https://srcdev.skatelescope.org/fts/rest davs://gfe02.grid.hep.ph.ic.ac.uk:2880/pnfs/hep.ph.ic.ac.uk/data/escape/ska/testing/da/06/ef1e45eb-009c-4501-923d-642dd3022c93 davs://amnesiac.cloud.cnaf.infn.it:8443/escape/ska/test-from-fts2
Job successfully submitted.
Job id: 83335d9e-dc37-11ec-b9ec-0242ac120004
[rbarnsley@src-dev install]$ fts-rest-transfer-status --access-token=${ACCESS_TOKEN} -s https://srcdev.skatelescope.org/fts/rest 83335d9e-dc37-11ec-b9ec-0242ac120004
Request ID: 83335d9e-dc37-11ec-b9ec-0242ac120004
Status: FINISHED
Client DN: 38d9270e-8d77-4e45-8ed0-b541be6c5e91
Reason:
Submission time: 2022-05-25T14:32:32
Priority: 3
VO Name: robbarnsleyAdmin (booking flights, writing up FTS work, updating outcomes for work with NALEDI team etc.)
Looking at FTS configuration
FTS deployment, auth not working.
Chat with Cedric/Martin about archiving of metadata.
Deployed FTS + rest + web monitoring. Writing up.
Some metadata stuff with Rizart (implementing it for Oracle).
Continued trying to deploy FTS3.
Sick.
Some debugging on rucio prod cluster.
Spoke to Rizart about s3 storage. Turns out that for conveyor, the protocol needs to be s3 in order for fts to sign the url (provided you've given it the secret) as there's no guarantee it will try it immediately and the signing has an expiration time.
For upload/download/deletion, the protocol must be https to tell gfal to sign the url before passing it to storage.
i.e. there's a protocol conflict depending on what operation you need to do, and whether it's the client or fts talking to storage.
Launched FTS server (src-fts3) and tried deployment.
Started looking at FTS.
Rucio meeting.
Prep + meeting.
Prep.